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-Jp'*  Abstract 


Significant  progress  was  made  In  a  number  of  aspects  of  nonlinear 
estimation  and  nonlinear  stochastic  systems .<^For  the  first  time,  the 
performance  of  suboptlmal  estimators  and  the  tTghtQess  of  estimation  lower 
bounds  for  a  nonlinear  estimation  problem  were  analyzed  by  comparison  with 
the  optimal  estimator,  lie  algebraic  techniques  were  applied  to  nonlinear 
estimation  problems;  these  have  provided  new  Insights  into  the-e^lstence  of 
recursive  finite  dimensional  estimators.  These  techniques  were  also 
applied  to  problems  In  system  Identification,  and  new  results  on  the\^ 
structure  of  these  problems  resulted.  For  certain  classes  of  Infinite 
horizon  decentralized  control  problems  Involving  Markov  chains,  the 
existence  of  a  stationary  optimal  policy  was  shown,  and  computational 
algorithms  for  such  problems  were  derived.  A  static  decentralized 
stochastic  control  problem  with  an  exponential  cost  functional  was  solved, 
and  this  resulted  In  the  only  nonquadratic  decentralized  stochastic  control 
problem  with  a  closed  form  optimal  policy.  In  addition,  the  optimal 
decentralized  gain  for  this  problem  was  expressed  as  an  explicit  projection 
of  the  optimal  centralized  law,  thus  leading  to  efficient  computational 
algorithms.  A  more  basic  study  of  decentralized  control  problems  led  to  a 
generalization  of  Radner’s  theorem  and  to  a  proof  of  global  optimality  of 
affine  policies  In  the  exponential  cost  problem.  A  dynamic  decentralized 
control  problem  with  exponential  cost  functional  was  solved,  and  an 
application  to  a  simplified  version  of  a  resource  allocation  problem 
associated  with  a  defense  network  was  pursued.  In  the  area  of  stochastic 
adaptive  control,  optimality  of  an  adaptive  control  was  proved  for  the 
first  time  In  the  literature  for  continuous-time  semi-Markov  decision 
problems;  In  particular,  problems  of  adaptive  control  of  service  and 
priority  assignment  were  solved.  Approved  for  public  release; 
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I.  SUMMARY  OF  RESEARCH  PROGRESS  AND  RESULTS 


During  the  five  years  of  research  supported  by  this  grant,  significant 
progress  has  been  made  in  a  number  of  aspects  of  nonlinear  estimation  and 
nonlinear  stochastic  systems.  In  this  section  this  progress  is  summarized, 
and  reference  is  made  to  the  resulting  publications  listed  in  Section  II. 

A.  Performance  of  Suboptimal  Estimators  and  Tightness  of  Bounds 

By  considering  the  class  of  continuous  time  nonlinear  estimation 
problems  for  which  we  have  previously  derived  finite  dimensional  optimal 
nonlinear  estimators,  we  have  In  [1]  and  [11],  for  the  first  time  In  the 
literature,  been  able  to  analyze  the  performance  of  suboptimal  estimators 
and  the  tightness  of  estimation  lower  bounds  for  a  nonlinear  system  by 
comparison  with  the  optimal  estimator.  A  system  for  which  we  can  construct 
the  optimal  estimator  was  studied;  the  optimal  estimator,  extended  Kalman 
filter  (EKF),  constant  gain  extended  Kalman  filter  (CGEKF),  best  linear 
estimator  (BLE),  and  Bobrovsky-Zakai  lower  bound  were  compared  both 
analytically  and  via  Monte  Carlo  simulations.  The  results  indicate  that 
the  performance  of  the  EKF  Is  as  good  as  that  of  the  optimal  estimator,  and 
that  the  Bobrovsky-Zakai  lower  bound  Is  tight  for  very  high  signal-to-noise 
ratio  but  very  ineffective  (i.e.,  "loose")  for  very  large  state  and 
observation  noises.  The  CGEKF  performs  almost  as  well  as  the  EKF  or 
optimal  filters;  thus  as  far  as  suboptimal  filter  design  for  such  systems 
is  concerned,  the  CGEKF  Is  probably  preferable  due  to  its  simple 
computational  requirements.  air  torce  office  of  scientific  research  >  afsc' 
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B.  Algebraic  and  Geometric  Methods  In  Estimation  and  Identification 


This  approach  was  pioneered  by  Brockett  and  Mitter,  but  we  have  been 
able  to  obtain  Interesting  results  for  much  wider  classes  of  nonlinear 
estimation  problems  by  using  these  methods.  The  approach  proceeds  by 
studying  the  (Zakal)  stochastic  partial  differential  equation  for  an 
unnormallzed  conditional  density  p  of  the  state  x  given  the  past 
observations  z: 

dp(t,x)  =  Lgp(t,x)dt  +  L<|P(t,x)dz,  (1) 

where  Lq  is  the  Fokker-Planck  operator  associated  with  x  and  is 

multiplication  by  h(x)  (the  observations  are  dz  =  h(x)dt  +  dv).  The  Lie 
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algebra  L  of  differential  operators  generated  by  Lq-%L^,  and  Lj  is  called 
the  estimation  Lie  algebra;  since  (1)  is  an  infinite  dimensional  bilinear 
equation  with  input  z,  It  Is  reasonable,  by  analogy  with  finite  dimensional 
control  systems,  to  study  the  algebra  L.  The  major  observation  of  Brockett 
is  that  If  a  recursive  finite  dimensional  estimator  for  some  conditional 
statistic  of  the  state  exists,  then  there  is  a  Lie  algebra  homomorphism 
from  L  to  the  Lie  algebra  F  of  the  finite  dimensional  filter.  Thus  we  have 
a  correspondence  between  statistics  which  are  finite  dimensionally 
recursively  computable  and  certain  homomorphisms  of  L  into  Lie  algebras  of 
vector  fields  on  finite  dimensional  manifolds. 

An  Important  step  In  this  program  has  been  completed  In  [22],  In  which 
we  analyze  in  detail  the  Lie  algebra  L  associated  with  the  Zakal  equation 
for  a  particular  class  of  nonlinear  filtering  problems  (a  class  for  which 
we  previously  derived  recursive  finite  dimensional  filters  for  all 
conditional  moments).  In  addition,  the  Lie  algebras  associated  with  the 
recursively  computable  conditional  moment  equations  are  computed,  and  the 


relationships  (e.g.,  homomorphlsms)  between  these  Lie  algebras  are  studied. 
This  study  provides  further  evidence  that  the  existence  of  finite 
dimensional  quotients  of  L  Is  closely  related  to  the  existence  of  recursive 
finite  dimensional  estimators;  In  fact,  this  Is  the  first  example  In  which  L 
is  an  Infinite  dimensional  Lie  algebra  for  which  such  a  study  has  been 
carried  out.  More  general  classes  of  nonlinear  estimation  problems  are 
considered  In  [13],  where  we  show  for  example  that  the  estimation  problem 
for  bilinear  systems  with  linear  observations  has  an  algebraic  structure 
similar  to  that  of  the  example  studied  In  [22].  On  the  other  hand,  we  show 
In  [13]  that  for  certain  classes  of  nonlinear  estimation  problems,  no 
conditional  statistic  of  the  state  can  be  computed  with  a  finite 
dimensional  filter.  This  Is  the  first  such  result  In  the  literature,  and 
It  applies  for  example  to  a  linear  system  with  cubic  observations  (the 
cubic  sensor  problem). 

We  proposed  application  of  these  Lie  algebraic  methods  to  the  problem 
of  simultaneous  state  and  parameter  estimation  In  linear  systems  with 
random  parameters  of  the  form 

dxt  *  A(p)xtdt +  b-|Utdt +b2dwt 

(2) 

dyt  =  <q,x^>  dt+  dv^ 

where  ut  Is  a  known  Input  function,  {w^}  and  { v^.}  are  Independent  Brownian 
motion  processes,  {x^}  and  {y^.}  are  respectively  the  state  and  observed 
process,  and  p  and  q  are  random  unknown  parameters.  For  example,  certain 
angle  modulation  problems  can  be  written  In  the  form  (2)  after  a  change  of 
coordinates.  In  [23]  we  have  analyzed  the  estimation  algebra  L  for  this 
problem  and  for  the  tracking  problem  In  which  p  and  q  are  random  processes; 
certain  tracking  problems  In  which  L  was  finite  dimensional  led  to  the 
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construction  of  finite  dimensional  filters.  This  problem  was  taken  one 
step  further  in  [16]  and  [19],  where  we  showed  that  the  estimation  algebra  L 
for  the  simultaneous  state  and  parameter  estimation  problem  (2)  is  a 
subalgebra  of  a  "current  algebra"  (current  algebras  have  also  recently 
begun  to  play  a  fundamental  role  in  certain  problems  in  physics).  It  then 
follows  that  L  is  embeddable  as  a  Lie  algebra  of  vector  fields  on  a  finite 
dimensional  manifold.  This  finite  dimensionality  is  manifested  in  our 
being  able  to  construct  a  finite  dimensional  recursive  filter  for  the 
conditional  density  of  the  parameters  evaluated  at  a  point.  In  addition, 
we  have  developed  a  Wei-Norman  type  procedure  for  the  computation  of  the 
joint  conditional  density  of  the  states  and  parameters,  thus  revealing  a 
set  of  functionals  of  the  observations  that  play  the  role  of  sufficient 
statistics  for  this  problem. 

In  related  work  on  parameter  estimation  in  linear  systems,  we  have 
defined  and  utilized  new  Riemannian  metrics  on  the  space  of  stable  linear 
systems.  If  one  wants  to  design  a  parameter  estimation  or  identification 
algorithm,  one  must  decide  upon  the  "cost"  of  identifying  the  wrong  values 
of  the  parameters;  this  is,  in  turn,  related  to  the  concept  of  distance 
between  models.  Since  the  class  of  stable  linear  systems  of  McMillan 
degree  n  forms  a  differentiable  manifold,  one  thus  needs  to  define  the 
distance  between  points  on  this  manifold,  which  is  precisely  what  a 
Riemannian  metric  provides.  In  [20],  we  have  defined  a  number  of 
Riemannian  metrics  on  this  manifold.  Computation  of  the  geodesics  of  the 
Riemannian  metrics  leads  to  new  insights  into  the  geometry  of  the  space  of 
stable  linear  systems.  In  addition,  Riemannian  metrics  are  used  to 
elucidate  the  structure  of  some  recursive  gradient  methods  of  parameter 
estimation. 


C.  Modeling  and  Approximation  of  Stochastic  Systems 

In  order  to  design  better  estimators*  one  must  also  obtain  a  better 
understanding  of  the  modeling  and  properties  of  nonlinear  stochastic 
systems.  In  [3],  we  have  pursued  this  goal  by  studying  the  modeling  and 
approximation  of  stochastic  differential  equations  driven  by  semimartingale 
noise  containing  both  continuous  and  jump  components.  The  stochastic 
calculus  and  stochastic  models  of  McShane  are  generalized  to  include  such 
differential  equations;  viewed  from  another  perspective*  we  have  defined 
for  the  first  time  the  analog  of  Stratonovich  stochastic  differential 
equations  when  the  noise  has  both  jump  and  continuous  components.  It  Is 
proved  that  the  models  defined  here  possess  the  same  desirable  properties 
(including  approximation  or  continuity  results)  as  those  of  McShane,  but 
they  are  applicable  to  a  much  wider  class  of  noise  processes.  The 
significance  of  this  work  is  related  to  the  fact  that,  for  models  to  be 
faithful  representations  of  physical  systems,  the  system  variables  should 
not  be  affected  significantly  by  small  changes  in  the  noi^e  processes 
driving  the  system.  The  results  presented  show  that  our  models  are,  under 
certain  hypotheses,  consistent  In  this  sense. 

D.  Decentralized  Static  and  Dynamic  Stochastic  Team  Problems 

We  have  been  Involved  In  the  study  of  decentralized  control  problems, 
which  have  many  applications  In  the  area  of  C  .  In  [6]  and  [14],  the 
stationary  conditions  of  Radner  are  shown  under  relaxed  hypotheses  to  be 
sufficient  to  establish  the  global  optimality  of  candidate  control  laws  for 
static  team  problems  with  convex  cost.  This  extension  of  Radner's  Theorem 
is  used  In  [2],  [6],  [12],  [14]  to  establish  the  global  optimality  of 
affine  laws  for  problems  with  positive  exponential  of  a  quadratic 
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performance  Index  with  jointly  Gaussian  state  and  observation  variables. 
Since  the  quadratic  cost  functional  results  in  the  only  previously  known 
closed  form  decentralized  optimal  control  policy,  these  results  for  the 
exponential  cost  functional  represent  an  important  generalization  (in  fact, 
the  quadratic  cost  results  are  a  special  case  of  these  as  a  certain 
parameter  approaches  zero). 

Team  problems  with  affine  control  laws  are  solved  as  constrained 
parameter  optimization  problems  in  [7].  This  methodology  eases  both 
notational  and  computational  difficulties.  This  approach  leads,  for  both 
the  quadratic  and  exponential  of  a  quadratic  cost  problems,  to  the 
representation  of  the  optimal  team  control  gains  as  explicit  projections  of 
the  optimal  centralized  gains.  This  representation  is  then  used  to  develop 
efficient  algorithms  for  the  solution  of  these  problems. 

In  addition,  we  have  studied  dynamic  team  problems  with  the 
exponential  of  a  quadratic  performance  index,  with  penalities  on  the 
terminal  state  and  all  Intermediate  control  values  [8],  [18].  For  a  system 
with  linear  dynamics  and  the  one-step  delayed  sharing  information  pattern, 
the  problem  reduces  via  dynamic  programming  to  a  succession  of  static  team 
problems  of  the  type  considered  above.  Moreover,  the  optimal  policy  of  the 
1th  team  member  at  time  k  Is  an  affine  function  of  both  the  one-step 
predicted  Kalman  filter  estimate  and  the  1th  team  member's  observation  at 
time  k.  Efficient  algorithms  are  developed  for  determining  the  gains  of 
this  affine  controller.  This  model  and  solution  are  applied  in  [8]  to  a 
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simplified  version  of  a  C  problem:  the  resource  allocation  problem 
associated  with  a  defense  network. 


E.  Decentralized  Control  of  Finite  State  Harkov  Processes  and  Multiaccess 
Broadcast  Communication 

A  different  approach  to  decentralized  control  has  been  taken  in  [4], 
[10],  [15].  In  this  work,  the  state,  control,  and  observations  all  take 
values  in  finite  sets;  the  state  is  a  controlled  finite  state  Harkov 
process.  Again,  the  one-step  delayed  sharing  information  pattern  is 
assumed.  Using  this  information  pattern,  the  dynamic  programming  algorithm 
Is  explicitly  carried  out  to  obtain  the  optimal  policy.  The  problems  are 
discussed  under  three  different  cost  criteria:  finite  horizon  problems 
with  total  expected  cost,  and  infinite  horizon  problems  with  discounted  or 
average  expected  cost.  Each  observation  is  noiseless  with  perfect 
recoverability  of  the  state  Information  from  all  the  observations. 

Improving  on  the  results  of  Sandell,  we  are  able  to  prove  the  existence  of 
stationary  (and  thus  much  more  easily  implemen table)  optimal  policies  for 
the  Infinite  horizon  problems. 

This  methodology  is  applied  in  [5],  [17]  to  multiaccess  broadcast 
communication  problems.  The  objective  of  multiaccess  broadcast 
communication  is  the  efficient  sharing  of  a  single  communication  medium 
among  many  users,  while  giving  each  access  to  the  full  bandwidth  of  the 
channel.  In  the  so-called  Aloha-type  schemes,  a  terminal  sends  a  new 
message  immediately  upon  receipt;  if  two  terminals  send  simultaneously,  a 
"collision*  occurs,  and  each  terminal  retransmits  after  a  random  delay. 

When  terminals  are  able  to  gather  some  information  about  the  state  of  the 
network,  feedback  control  policies  (centralized,  decentralized,  or 
adaptive)  may  be  Implemented  to  Improve  performance.  A  number  of  such 
control  schemes  have  been  proposed  in  the  literature.  On  the  other  hand, 
this  problem  fits  into  the  dec  ntralized  control  framework  discussed  in 


[4].  In  [5],  we  have  applied  the  optimal  decentralized  control  policies 
derived  in  [4]  to  problems  of  this  type,  and  compared  the  results  with 
those  of  previously  proposed  algorithms.  Tradeoffs  between  the  information 
available  to  each  terminal  and  the  value  of  the  performance  index  were 
examined  in  detail. 


F.  Stochastic  Adaptive  Control  of  Semi-Markov  Decision  Processes 
In  [9]  and  [21],  we  have  considered  the  problem  of  determining 
adaptive  policies  for  the  optimal  control  of  average  cost  semi -Markov 
decision  processes  with  transition  probabilities  and  sojourn  time 
distributions  depending  on  unknown  parameters.  To  solve  the  problem,  an 
estimation  and  control  scheme  was  proposed  and  its  optimality  shown  under 
appropriate  recurrency  conditions.  Our  approach,  which  is  essentially  that 
of  the  "naive  feedback  controller"  in  stochastic  control  theory,  differs 
from  previous  results  for  discrete-time  Markov  decision  processes  in  that 
we  consider  continuous-time  (not  necessarily  Markov)  processes,  and  the 
sampling  times  in  our  parameter  estimation  scheme  are  random,  as  opposed  to 
synchronous  sampling.  In  particular,  motivated  by  problems  in  computer 
networks,  we  have  considered  in  [21]  the  priority  assignment  (or  dynamic 
scheduling)  problem  for  a  single-server  queueing  system  with  several 


classes  of  customers  who  arrive  according  to  Independent  Poisson  processes. 
The  service  rates  are  considered  constant  but  unknown,  and  the  problem  is 
to  decide,  at  the  completion  of  each  service  and  given  the  state  of  the 
system,  which  class  to  admit  next  into  service.  Given  a  linear  cost 
structure,  the  objective  is  to  minimize  the  long  run  average  cost  per  unit 
time.  An  adaptive  control  scheme  was  proposed,  and  we  proved  its 
optimality.  A  similar  method  was  used  on  certain  problems  of  adaptive 
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control  of  service  rates  in  [9].  In  both  [9]  and  [21],  the  approach  was  to 
show  an  equivalence  between  the  continuous-time  decision  problem  and  a 
corresponding  discrete-time  problem,  prove  convergence  of  the  parameter 
estimates  for  the  discrete-time  problem,  and  then  to  prove  that  the  cost  of 
using  the  adaptive  policy  is  the  same  as  that  for  the  optimal  policy  with 


known  parameters. 
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